Motion compensation and motion estimation leveraging a continuous coordinate system

ABSTRACT

Computer processor hardware receives settings information for a first image. The first image includes a set of multiple display elements. The computer processor hardware receives motion compensation information for a given display element in a second image to be created based at least in part on the first image. The motion compensation information indicates a coordinate location within a particular display element in the first image to which the given display element pertains. The computer processor hardware utilizes the coordinate location as a basis from which to select a grouping of multiple display elements in the first image. The computer processor hardware then generates a setting for the given display element in the second image based on settings of the multiple display elements in the grouping.

RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.13/893,677 entitled “MOTION COMPENSATION AND MOTION ESTIMATIONLEVERAGING A CONTINUOUS COORDINATE SYSTEM” (Attorney Docket No.VNO12-(09), filed on May 14, 2013, the entire teachings of which areincorporated herein by this reference.

U.S. patent application Ser. No. 13/893,677 is related to and claims thebenefit of U.S. Provisional Patent Application Ser. No. 61/646,797entitled “SIGNAL ENCODING, DECODING AND RECONSTRUCTION OF TIME-BASEDAND/OR MULTIDIMENSIONAL SIGNALS BASED ON MULTIDIMENSIONAL TIER-BASEDINHERITANCE” (Attorney Docket No. VNO12-01p), filed on May 14, 2012, theentire teachings of which are incorporated herein by this reference.

U.S. patent application Ser. No. 13/893,677 is related to and claims thebenefit of U.S. Provisional Patent Application Ser. No. 61/647,426entitled “ESTIMATION, ENCODING, DECODING AND USAGE OF MOTION INFORMATIONIN MULTIDIMENSIONAL SIGNALS THROUGH MOTION ZONES, MOTION MATRIXES, WARPMAPS AND MOTION TRANSFORMS” (Attorney Docket No. VNO12-(02p), filed onMay 15, 2012, the entire teachings of which are incorporated herein bythis reference.

This application is also related to U.S. patent application Ser. No.13/188,188 entitled “INHERITANCE IN A TIERED SIGNAL QUALITY HIERARCHY,”(Attorney Docket No. VNO11-00), filed on Jul. 21, 2011, the entireteachings of which are incorporated herein by this reference.

This application is related to U.S. patent application Ser. No.13/188,201 entitled “TIERED SIGNAL DECODING AND SIGNAL RECONSTRUCTION,”(Attorney Docket No. VNO11-01), filed on Jul. 21, 2011, the entireteachings of which are incorporated herein by this reference.

This application is related to U.S. patent application Ser. No.13/188,207 entitled “SIGNAL PROCESSING AND TIERED SIGNAL ENCODING,”(Attorney Docket No. VNO11-02), filed on Jul. 21, 2011, the entireteachings of which are incorporated herein by this reference.

This application is related to U.S. patent application Ser. No.13/188,220 entitled “UPSAMPLING IN A TIERED SIGNAL QUALITY HIERARCHY,”(Attorney Docket No. VNO11-03), filed on Jul. 21, 2011, the entireteachings of which are incorporated herein by this reference.

This application is related to U.S. patent application Ser. No.13/188,226 entitled “SIGNAL PROCESSING AND INHERITANCE IN A TIEREDSIGNAL QUALITY HIERARCHY,” (Attorney Docket No. VNO11-04), filed on Jul.21, 2011, the entire teachings of which are incorporated herein by thisreference.

This application is related to U.S. patent application Ser. No.13/352,944 entitled “SIGNAL ANALYSIS AND GENERATION OF TRANSIENTINFORMATION,” (Attorney Docket No. VNO11-05), filed on Jan. 18, 2012,the entire teachings of which are incorporated herein by this reference.

This application is related to U.S. Provisional Patent Application Ser.No. 61/563,169 entitled “TIER-BASED SYSTEM TO SEPARATE AMULTIDIMENSIONAL SIGNAL INTO STABLE/PREDICTABLE INFORMATION ANDTRANSIENT INFORMATION,” (Attorney Docket No. VNO11-05p), filed on Nov.23, 2011, the entire teachings of which are incorporated herein by thisreference.

This application is related to U.S. patent application Ser. No.13/188,237 entitled “TRANSMISSION OF RECONSTRUCTION DATA IN A TIEREDSIGNAL HIERARCHY,” (Attorney Docket No. VNO11-06), filed on Jul. 21,2011, the entire teachings of which are incorporated herein by thisreference.

This application is related to U.S. Provisional Patent Application Ser.No. 61/558,302 entitled “UPSAMPLING AND DOWNSAMPLING OF MOTION MAPS ANDOTHER AUXILIARY MAPS IN A TIERED SIGNAL QUALITY HIERARCHY,” (AttorneyDocket No. VNO11-07p), filed on Nov. 10, 2011, the entire teachings ofwhich are incorporated herein by this reference.

This application is related to U.S. patent application Ser. No.13/303,554 entitled “UPSAMPLING AND DOWNSAMPLING OF MOTION MAPS ANDOTHER AUXILIARY MAPS IN A TIERED SIGNAL QUALITY HIERARCHY,” (AttorneyDocket No. VNO11-07), filed on Nov. 23, 2011, the entire teachings ofwhich are incorporated herein by this reference.

This application is related to U.S. Provisional Patent Application Ser.No. 61/587,989 entitled “DISTINCT ENCODING/DECODING OFSTABLE/PREDICTABLE INFORMATION AND TRANSIENT/STOCHASTIC INFORMATION,”(Attorney Docket No. VNO11-09p), filed on Jan. 18, 2012, the entireteachings of which are incorporated herein by this reference.

This application is related to U.S. patent application Ser. No.13/744,808 entitled “DISTINCT ENCODING AND DECODING OF STABLEINFORMATION AND TRANSIENT/STOCHASTIC INFORMATION” (Attorney Docket No.VNO11-09), filed on Jan. 18, 2013, the entire teachings of which areincorporated herein by this reference.

BACKGROUND

Many techniques are known in the art to deal with compression anddecompression of multidimensional signals or of signals evolving alongtime. This is the case of audio signals, video signals and othermultidimensional signals like volumetric signals used in scientific andmedical areas.

In order to achieve high compression ratios, those techniques exploitthe spatial and time correlation inside the signal. Conventional methodsidentify a reference and try to determine the difference of the signalbetween a current location and the given reference. This is done both inthe spatial domain, where the reference is a portion of already receivedand decoded spatial plane, and in the time domain, where a singleinstance in time of the signal (e.g., a video frame in a sequence offrames) is taken as a reference for a certain duration. This is thecase, for example, of MPEG-family compression algorithms, wherepreviously-decoded macro blocks are taken as reference in the spatialdomain and I-frames and P-frames are used as reference for subsequentP-frames in the time domain.

Known techniques exploit spatial correlation and time correlation inmany ways, adopting several different techniques in order to identify,simplify, encode and transmit differences. In accordance withconventional methods, in order to leverage on spatial correlation adomain transformation is performed (for example into a frequency domain)and then lossy deletion and quantization of information is performed. Inthe time domain, instead, conventional methods transmit the quantizeddifference between the current sample and a motion-compensated referencesample.

In order to maximize the similarity between samples, encoders try toestimate the modifications along time occurred vs. the reference signal.This is called, in conventional encoding methods (e.g., MPEG familytechnologies, VP8, etc.), motion estimation and compensation.

Motion information is transmitted to the decoder in order to enablereconstruction of the current sample by leveraging information alreadyavailable at the decoder for the reference sample: in state-of-the-artmethods this is done using motion vectors at a macro block basis. Inother words, a motion vector can indicate motion at a block levelincluding multiple display elements.

Traditionally, motion information has been represented by means ofoffset motion vectors, i.e., vectors indicating the position of asimilar portion of a picture (e.g., a block of plane elements, or“pels”, often called picture elements or “pixels” for the case of 2Dimages) in a reference picture. For example, as discussed above, usingblock motion compensation (BMC), the images of a video sequence can bepartitioned into blocks of pixels. Each block B in a current image canbe predicted based on a block B₀ of equal size in a reference frame. Theposition of the block B₀ in the reference image with respect to theposition of B in the current image can be encoded as an offset motionvector. In such cases, the motion vector indicates the opposite of theestimated x and y movement of the block of pixels (in particular, itindicates the opposite of the movement since it points from B to B₀,while the movement is from B₀ to B).

A motion vector is typically encoded with sub pixel precision (i.e., canspecify movements also of fractions of a pixel) because the encoderwants to be able to capture also subtle movements of less than a fullpixel. According to MPEG family codecs, the blocks are not transformedother than being shifted to the position of the predicted block, andadditional information must be encoded through residual data indicatingdifferences between block B₀ and block B.

Motion estimation is typically referred to as the process of determiningmotion vectors that suitably describe the transformation from onepicture to another, usually from adjacent frames in a video sequence.Motion estimation is typically based on an assumption that image values(brightness, color, etc., expressed in a suitable color space) remainconstant over time, though their position in the image may change. Theunderlying assumption of motion estimation through motion vectors isthat the possible movements of the portion of the image identified bythe motion vector (e.g., macro-block) are limited to translationalmovements.

In state of the art technologies, coordinates of motion vectorsassociated to either a pel or a group of pels are expressed based on adiscrete coordinate system (i.e., with a finite set of symbols), eitherpossessing step width of the same resolution as the current image (“pelresolution”, i.e., current image and reference image have the sameresolution) or possessing sub-pel resolutions (e.g., by way ofnon-limiting examples, 1/14^(th) of a pel, 1/18^(th) of a pel, etc.). Inthis last case, the reference image has a higher resolution than thecurrent image, in order to allow a motion vector to point to a givenposition with sub-pixel resolution (with respect to the resolution ofthe current image); essentially, the reference image is supersampledwith a given scale factor, and the coordinates of motion vectors areexpressed with integer numbers in the coordinate system of thesupersampled reference image. In other words, even though a display doesnot have the ability to display such a high resolution, a supersampled(high-resolution) rendition of an image is produced for a givenreference image, just to support motion compensation operations. Motionvectors can be used to identify which portion of the rendition of theimage is to be used to reconstruct a display signal.

Leveraging motion vectors with sub-pel resolution allows for betterprecision in motion estimation and in motion compensation, but alsoimplies the significant disadvantage of requiring a higher amount ofmemory at the decoder side, since the buffer that stores the “super highresolution” of the reference image needs to store a much higher numberof pels than the number that it is necessary to display on a respectivedisplay screen.

Known encoding techniques based on block motion compensation and onoffset motion vectors using integer coordinates (i.e., coordinates withfixed precision, such as ⅛^(th) of a pixel) have several importantdrawbacks, suitably addressed by novel methods described herein. Mostnotably, the use of offset coordinates with a given sub-pixel precisiontypically requires to buffer an upsampled rendition of the referenceimage at the given sub-pixel resolution: as a consequence, capturingvery subtle movements (e.g., 1/128 of a pixel, important for instance inthe case of high frame-rate video signals or in the case of complexmovements such as a 1% zoom with 2-degree rotation) is not feasible dueto memory limitations and to the high amount of computations that wouldbe necessary to calculate the supersampled reference image. Generationand processing of a super high-resolution reference image is undesirablefor a number of reasons.

BRIEF DESCRIPTION

Embodiments herein deviate with respect to conventional systems andmethods, providing novel methods to estimate, encode and leverage motioninformation so as to generate a suitable prediction of a current image(or “target image”) based on motion compensation of a reference image,hence supporting methods such as motion estimation, motion compensation,signal encoding, signal quality enhancement (e.g., denoising,super-resolution, etc.), signal interpolation (e.g., increase offrame-rate), special effects, computer graphics, medical imaging,computer vision, augmented reality applications, etc.

One embodiment herein includes a method for calculating or predictingthe value of an element of a target image based on the value of anarbitrary location of a reference image, whereby such arbitrary locationis expressed with fractional coordinates (such as floating pointnumbers, high-precision fixed-point numbers, real numbers, non-integernumbers, etc.) independent of the actual sample grid (i.e., resolution)of the reference image used as a basis to reconstruct a rendition of theimage. In contrast to pre-calculating and producing a supersampledreference image at a higher resolution and then performing motioncompensation by means of an integer coordinate system based on thesupersampled grid, certain embodiments illustrated herein do not need topre-calculate any supersampled rendition of the reference image andinstead calculate reference elements on-the-fly at any arbitrarylocation in the reference image (e.g., without limitation, viaon-the-fly resampling techniques).

According to embodiments herein, motion compensation can be effectivelyimplemented using a substantially continuous coordinate system andeffectively supporting motion compensation of very subtle movements(e.g., if necessary, even smaller than 1/10^(th) of a pel) withrelatively limited buffer memory cost and computational cost.Essentially, motion compensation according to embodiments herein allowsfor extremely high precision (e.g., capturing movements of less than1/100^(th) of a pel) and may leverage resampling operations performed onthe fly, without the need of storing large reference images at higherresolutions.

Embodiments herein may be useful in conjunction with traditional motioncompensation approaches, and may be even more useful in conjunction withmotion zones and motion matrixes (as opposed to block motioncompensation with offset motion vectors), as described in relatedapplications.

For simplicity, non-limiting embodiments illustrated herein refer to asignal as a sequence of multi-dimensional samples (i.e., sets of one ormore elements organized as arrays with one or more dimensions, e.g., byway of non-limiting example sets of picture elements organized astwo-dimensional images) occurring at a given sample rate along the timedimension. In the description the terms “image” or “plane” (intendedwith the broadest meaning of “hyperplane”, i.e., array of elements withany number of dimensions and a given sampling grid) will be often usedto identify the digital rendition of a sample of the signal along thesequence of samples, wherein each plane has a given resolution for eachof its dimensions (e.g., X and Y), and comprises a set of plane elements(or “element”, or “pel”, for two-dimensional images often called“pixel”, for volumetric images often called “voxel”, etc.) characterizedby one or more “values” or “settings” (e.g., by ways of non-limitingexamples, color settings in a suitable color space, settings indicatingdensity level, settings indicating temperature levels, settingsindicating audio pitch, settings indicating amplitude, etc.). Each planeelement is identified by a suitable set of coordinates, indicating theinteger positions of the element in the sampling grid of the image.

As non-limiting examples, a signal can be an image, an audio signal, amulti-channel audio signal, a video signal, a multi-view video signal(e.g., 3D video), a volumetric signal (e.g., medical imaging, scientificimaging, holographic imaging, etc.), a volumetric video signal, or evensignals with more than four dimensions.

Embodiments illustrated herein will be particularly focused on signalsevolving over time and featuring some degree of motion from one sampleto the next, i.e., samples are time correlated. Also very high samplerates (e.g., also over 1,000 images per second, the motion of which istypically badly described by conventional motion estimation andcompensation methods) are easily addressed by the described embodiments.

For simplicity, non-limiting embodiments illustrated herein often referto signals that are displayed as sequences of 2D planes of settings(e.g., 2D images in a suitable color space), such as for instance avideo signal. However, people skilled in the art can easily understandthat the same concepts and methods are also applicable to any othertypes of time-based signal (e.g., multi-view video signals, 3D videosignals, sequences of 3D volumetric signals, etc.), and also tonon-time-based multi-dimensional signals (e.g., sequences of audiochannels of a same audio signal, two-dimensional pictures,volumetric/holographic images, plenoptic images, etc.). As anon-limiting example of a non-time-based signal that can benefit ofnovel compensation methods described herein, a series of two-dimensionalslices of a CAT-scan or an MRI (i.e., a non-time-based three-dimensionalsignal) can be suitably represented as a series of two-dimensionalsamples along a dimension (i.e., the axis along which the slices weretaken), and processed according to methods illustrated herein, as if theaxis along which the slices were taken was a time dimension (by assumingeither a constant sample rate or even a variable sample rate).

In a non-limiting embodiment described herein, a signal processor isconfigured to calculate (“predict”) compensated settings for elements ofan image (“destination image”, or “compensated image”) leveragingon-the-fly resampling operations in order to access any arbitraryposition (x¹, y¹) of a reference image, regardless of the actualresolution (i.e., sampling grid) of the reference image. In this way, itis possible to leverage on a coordinate system where each coordinate isexpressed with arbitrary precision (e.g., without limitation, even by afloating point number or a high precision fixed-point number, as opposedto an integer number in the coordinate system of the reference image),so that the resolution of the reference image is treated as essentiallyinfinite/continuous (“continuous coordinates”, or “fractionalcoordinates”). As already mentioned, this approach is extremelyinnovative since state-of-the-art encoding and decoding techniques havebeen based so far on the assumption that the reconstructed signal andreference signals have a finite resolution, with each coordinateindicating an element comprised in a discrete grid/set of elements.

In a non-limiting embodiment, on-the-fly resampling operations areperformed by selecting a set of elements of the reference imagebelonging to the sampling grid of the reference image and close to thearbitrary position indicated by the fractional coordinates of the motionvector. According to the chosen resampling method (e.g., by way ofnon-limiting example, bicubic resampling), the signal processor selectsan appropriate number of elements (e.g., without limitation, the 16elements with the closest center to the arbitrary location, or fewerelements if the arbitrary location is close to the borders of thereference image) and calculates the weights to apply to each element.Lastly, the signal processor calculates the sampling value to assign tothe arbitrary position by performing a weighted average of the selectedelements.

Some non-limiting embodiments described herein use continuouscoordinates (e.g., by way of non-limiting example, by representingcoordinates with floating point numbers) and transform matrixes (asopposed to simple offset motion vectors) in order to describe movementsof groupings of elements, notably increasing precision in describing theactual movements. Motion estimation and compensation are often criticaloperations in order to achieve high compression ratios. Performingprecise motion compensation provides better predictions and thus lowerentropy residuals. In prior art methods, motion estimation andcompensation in video encoding and video processing have been limited topel areas with translation movements (typically expressed by means ofoffset motion vectors), which tends to be a limiting and low qualityapproach. In fact, objects are affected by a much more complex set ofpossible movements. In order to capture this complexity, non-limitinginnovative embodiments described herein model motion by using transformmatrixes rather than motion vectors. Movements like rotations, skews orzooms can be described using affine transforms and homogeneouscoordinates. Using higher order matrixes (e.g., projective matrixes),also perspective changes can be described. Usage of transform matrixesin a signal encoding (e.g., video compression) domain is veryinnovative, and entails a number of consequences that notablydistinguish novel embodiments described herein from conventionalmethods.

A non-limiting embodiment leverages transform matrixes and homogenouscoordinates in order to represent complex movements including—withoutlimitation—zoom, rotation and offset. In such embodiment, the signalprocessor performing motion compensation calculates for every givenelement with integer coordinates (x, y) of the destination image thecorresponding location with continuous coordinates (x¹, y¹) in thereference image to leverage for motion compensation by multiplying thecoordinates of the given element expressed in homogenouscoordinates—i.e., (x, y, 1)—by an affine transform matrix. In this waythe signal processor essentially calculates an offset motion vector withfractional coordinates for every element of the destination image. Thesignal processor then calculates the motion-compensated value to assignto the given element (x, y) by performing a weighted average of selectedelements of the reference image, whereby both the elements and theweights in the weighted average depend at least in part on location (x¹,y¹).

Another non-limiting embodiment leverages projective transform matrixes,so as to represent even more complex movements. A non-limitingembodiment, in performing the necessary calculations for motioncompensation, takes advantage of the modern hardware used in gaming and3D rendering, so as to exploit continuous-coordinate motion compensationat very limited computational cost. Modern hardware can performinterpolations on the fly (e.g., via on-the-fly resampling) by usingfloat coordinates for the computed element. One of the advantagesassociated with the usage of fractional coordinates and on-the-flyresampling is the possibility to represent very subtle movements whileat the same time reducing memory usage at both the encoder and thedecoder side. Motion estimation and motion compensation rely on theresampling operations performed on the fly, without any need forgenerating and storing large reference images at higher resolutions.

Using continuous coordinates is very important when motion compensationis based on motion matrixes (i.e., more sophisticated movements than asimple translation), because sophisticated movements often requireextremely fine sub-pixel resolution, not achievable with the standardtechnique of supersampling the reference image (e.g., withzoom/divergence, even levels of zoom as small as 1%—i.e., coordinatemultiplications by 0.01—are relevant).

In a non-limiting embodiment, motion of specific portions of a signal isexpressed by means of parameters corresponding to rotation, scaling,translation and shear mapping. Or, equivalently, with an affinetransform matrix such as the following applied to a vector (x, y, 1) inhomogenous coordinates—i.e., (x, y, w) with w normalized to 1:

$M = \begin{bmatrix}a & b & 0 \\c & d & 0 \\e & f & 1\end{bmatrix}$

In other non-limiting embodiments, motion information is expressed byusing projective transforms, i.e., 3×3 matrix with 8 relevantcoefficients, and the 9^(th) coefficient normalized to 1, thusdescribing with a single transform scaling, rotation, offset, shearingand perspective change. Since some of such transforms require a divisionoperation for each transform, a non-limiting embodiment uses approximatedivision operations (e.g., by way of non-limiting examples, using only16 bits, or using some of the algorithms commonly used for shaders).

Motion matrixes require to send to the decoder a higher number ofparameters representing motion with respect to the number of parametersrequired for simple offset motion vectors: as a consequence, thebenefits of using motion matrixes is higher when they are applied torelatively large and arbitrarily-shaped groupings of elements (“motionzones”), e.g. representing an object moving in a consistent way.

In a non-limiting embodiment illustrated herein, a signal processorconfigured as an encoder receives a current (target) image and areference image, performs motion estimation and identifies in thecurrent image one or more motion zones (arbitrary—contiguous ornon-contiguous—portions of the signal) and corresponding descriptiveinformation on the motion of each motion zone, the motion beingexpressed in a continuous coordinate system. In a non-limitingembodiment, the encoder decides the maximum number of motion zones basedon a set of parameters (e.g., by way of non-limiting example, availablecomputational power, target encoding latency, target compressionefficiency, etc.).

In another non-limiting embodiment illustrated herein, a signalprocessor configured as a decoder receives motion zone information(e.g., a motion zone map) and then receives descriptive information onmotion with the motion characteristic of each motion zone (e.g., by wayof non-limiting embodiment, by receiving a set of parameterscorresponding to a motion matrix for each motion zone). Based at leastin part on the motion zone map and on descriptive information on themotion of each motion zone, for each element of the target image thedecoder calculates a motion vector, the coordinates of the motion vectorbeing expressed in a continuous coordinate system (e.g., withoutlimitation, by means of floating point numbers). Based on the motionvectors, reference values in arbitrary locations are fetched from areference image via on-the-fly resampling, allowing for motioncompensation with higher precision than traditional approaches based onfixed grids of elements and integer-based coordinates.

In a non-limiting embodiment, the signal processor produces amotion-compensated image with a different number of elements than thereference image. In another non-limiting embodiment, one or moreelements of the motion-compensated images are assigned a default value(e.g., “N/A value”, for instance—without limitation—wherein thecorresponding location in the reference image is outside of the boundaryof the image or is itself characterized by an “N/A value”, or whereindescriptive information on motion indicates that the specific elementscannot be predicted by means of motion compensation of the referenceimage).

In accordance with further non-limiting embodiments, the input signal isencoded and decoded by means of a tier-based hierarchical encodingmethod, and motion compensation with continuous coordinates is leveragedwithin a tier-based hierarchical encoding loop.

In accordance with one embodiment, computer processor hardware: receivesreference image information, the reference image information defining agrid of multiple elements at a given resolution; receives compensationinformation for an image element in a compensated image, settings ofdisplay elements in the compensated image derived at least in part fromthe reference image information; processes the received compensationinformation to produce a set of coordinates indicating a correspondingoff-grid location of the reference image (such as a location in thereference image that is comprised in between two sampling positions ofthe sampling grid of the reference image); calculates a value for thecorresponding off-grid location of the image element based on a group ofmultiple elements in the grid; and assigns the value to the imageelement in the compensated image.

In one embodiment, the coordinates that indicate the correspondingoff-grid location are expressed with a substantially higher resolutionthan the given resolution of the grid.

Calculating the value can include: applying an algorithm to identifywhich of the multiple elements in the grid to include in the group, thegroup of multiple elements disposed in a vicinity of the correspondingoff-grid location. In one embodiment, the algorithm applies one or moremathematical operations to settings of the group of multiple elements toderive the value for the display element being reconstructed.

In accordance with further embodiments, the set of coordinates indicatesthe corresponding off-grid location is expressed via numbersrepresenting quantities in a real domain. The set of coordinatesindicating the corresponding off-grid location can have a sufficientlyhigh resolution to specify an offset with respect to on-grid locationsin the grid by less than 1/32nd of a display element in the grid.

The resolution of the reference image information can be substantiallythe same as a resolution of the compensated image.

Calculating the coordinates of the off-grid location can includeapplying floating point operations to numbers in floating point formatrepresenting quantities in a real domain.

Calculating the value can include applying floating point operations tonumbers in floating point format representing quantities in a realdomain.

Calculating the coordinates of the off-grid location can includeapplying mathematical operations with numbers in fixed-point formatrepresenting quantities in a real domain.

Calculating the value can include applying mathematical operations withnumbers in fixed-point format representing quantities in a real domain.

The received compensation information can comprise one or moreparameters expressed by numbers representing quantities in a realdomain.

In one embodiment, the computer processor hardware calculating the valuefor the display element executes a shader in a Graphics Processing Unit.

As discussed herein, calculating the value for a respective displayelement can include executing at least one resampling operation based onthe calculated respective location in the reference image.

Processing the received compensation information can include: applyinglinear combinations corresponding to coefficients of a motion matrix toparameters corresponding to coordinates of a respective image element toproduce the corresponding set of coordinates in the grid of thereference image; and applying linear combinations corresponding tocoefficients of a same motion matrix to produce a respective set ofcoordinates for each of multiple display elements in the compensatedimage.

In accordance with further embodiments, processing the receivedcompensation information can include: calculating a vector in homogenouscoordinates corresponding to the set of coordinates associated with theimage element in the compensated image; and multiplying the vector by amatrix, wherein the coefficients in the matrix are specified at least inpart based on the compensation information.

In yet further embodiments, the computer processor hardware calculatesthe set of coordinates as coordinates (x^(T), y^(T)), the coordinates(x^(T), y^(T)) specifying the off-grid location of the image element,the image element residing at location (x, y) in the compensated image,the coordinates (x^(T), y^(T)) calculated by performing sums andmultiplications of x and y with six coefficients a, b, c, d, e and f,according to the following formula: [x^(T) y^(T)]=[a·x+c·y+e b·x+d·y+f].

In yet further embodiments, the computer processor hardware calculatesthe set of coordinates as coordinates (x^(T), y^(T)), the coordinates(x^(T), y^(T)) specifying the off-grid location of the image element,the image element residing at location (x, y) in the compensated image,the coordinates (x^(T), y^(T)) calculated by performing sums andmultiplications of x and y with eight coefficients a, b, c, d, e, f, gand h, according to the following formula:

$\begin{matrix}\left\lbrack x^{T} \right. & \left. y^{T} \right\rbrack\end{matrix} = \left\lbrack \begin{matrix}\frac{{a \cdot x} + {c \cdot y} + e}{{g \cdot x} + {h \cdot y} + 1} & {\left. \frac{{b \cdot x} + {d \cdot y} + f}{{g \cdot x} + {h \cdot y} + 1} \right\rbrack.}\end{matrix} \right.$

In accordance with further embodiments, at least one element in thecompensated image is not contiguous with a set of other elements in thecompensated image for which respective coordinates (x^(T), y^(T)) arecalculated according to the same parameters (such as a same motionmatrix). The compensation information can include one or more parameterscorresponding to motion types selected from the group comprising:scaling, rotation and offset. The one or more parameters of compensationinformation can be used to generate one or more motion vectors fordisplay elements in the compensated image. In yet other embodiments,compensation information can include one or more parameterscorresponding to motion types selected from the group comprising:scaling, rotation, offset, shearing, perspective change.

Further embodiments herein include associating a default value (such asa “Not available” default value) to a respective set of coordinates thatspecify a location outside of the grid and that are located with adistance from a closest sample of the grid beyond a given thresholdvalue.

The signal or compensated image being reconstructed can be of anysuitable type such as a two-dimensional image, a volumetric image, amono-dimensional image, a multi-dimensional image, etc.

In accordance with yet another embodiment, computer processor hardware:receives settings information for a first image, the first imageincluding a set of multiple display elements; receives motioncompensation information for a given display element in a second image,the motion compensation information indicating a coordinate locationwithin a particular display element in the first image to which thegiven display element of the second image pertains; utilizes thecoordinate location as a basis from which to select a grouping ofmultiple display elements in the first image; and generates a settingfor the given display element in the second image based on settings ofthe multiple display elements in the grouping.

In this latter embodiment, generating the setting for the given displayelement can include: calculating the setting for the given displayelement based at least in part on distances of the multiple displayelements in the grouping with respect to the coordinate location in theparticular display element.

Generating the setting for the given display element can include:calculating the setting for the given display element based at least inpart on settings of the multiple display elements in the grouping.

The coordinate location as specified by motion information can representa centroid from which to generate the setting of the given displayelement. Generating the setting for the given display element caninclude: producing weighted values (such as the coefficients of aresampling kernel), the weighted values varying depending on arespective distance between the centroid of the given display elementand a respective centroid of an element in the grouping; and applyingthe weighted values to settings of the display elements in the groupingto produce the setting for the given display element.

In one embodiment, the coordinate location as specified by the motioninformation can specify a location other than a center of the particulardisplay element in the first image. Substantially, the coordinatelocation can specify any location in the first image, as with acontinuous coordinate system with arbitrary precision.

As mentioned, in one embodiment, the resolution of the first image canbe substantially equal to a resolution of the second image. Inaccordance with further embodiment, the resolution of the first imagecan be substantially equal to a resolution of the second image, but witha larger field of view (i.e., the first image substantially comprisesmore information than the second image). In accordance with furtherembodiments, the resolution of the first image (or reference image) canbe substantially different than a resolution of the second image(reproduced image based on reference image). For instance, the firstimage can be a predictor image at a lower level of quality in a tieredtemporal hierarchy, with a lower resolution than the second image.

In accordance with further embodiments, utilizing the coordinatelocation as a basis from which to select the grouping of multipledisplay elements in the first image can include: creating the groupingof multiple display elements to include the particular display elementand a display element in the first image located in a vicinity of theparticular display element; and interpolating the setting of the givendisplay element based on: i) a proximity of the coordinate location withrespect to a centroid of the particular display element and a centroidof the display element in the vicinity of the particular displayelement, ii) a setting of the particular display element, and iii) asetting of the display element located in the vicinity of the particulardisplay element.

In still further embodiments, the first image is a reference image; theset of multiple display elements represents at least a portion of anobject present in the first image.

As further described herein, the coordinate location as specified by amotion vector can represent an offset of the given display element withrespect to a centroid of the particular display element by a fraction ofa display element greater than zero but less than 1.

Note that embodiments herein may be implemented in software or hardware,or may be implemented using a combination of software and hardware, andcan include a configuration of one or more computerized devices,routers, network, workstations, handheld or laptop computers, tablets,mobile phones, game consoles, set-top boxes, etc., to carry out and/orsupport any or all of the method operations disclosed herein. In otherwords, one or more computerized devices or processors can be programmedand/or configured to operate as explained herein to carry out differentembodiments.

In addition to the techniques as discussed above, yet other embodimentsherein include software programs to perform the steps and operationssummarized above and disclosed in detail below. One such embodimentcomprises a computer-readable, hardware storage resource (i.e., anon-transitory computer readable media) including computer programlogic, instructions, etc., encoded thereon that, when performed in acomputerized device having a processor and corresponding memory,programs and/or causes the processor to perform any of the operationsdisclosed herein. Such arrangements can be provided as software, code,and/or other data (e.g., data structures) arranged or encoded on acomputer readable medium such as an optical medium (e.g., CD-ROM,DVD-ROM or BLU-RAY), flash memory card, floppy or hard disk or any othermedium capable of storing computer readable instructions such asfirmware or microcode in one or more ROM or RAM or PROM chips or as anApplication Specific Integrated Circuit (ASIC). The software or firmwareor other such configurations can be installed onto a computerized deviceto cause the computerized device to perform the techniques explainedherein.

Accordingly, one particular embodiment of the present disclosure isdirected to a computer program product that includes a computer-readablehardware storage medium having instructions stored thereon forsupporting signal processing operations.

One or more embodiments herein include a computer readable storagemedium and/or system having instructions stored thereon. Theinstructions, when executed by computer processor hardware of one ormore computer devices, causes the computer processor hardware to performoperations of: receiving reference image information, the referenceimage information defining a grid of multiple elements at a givenresolution; receiving compensation information for an image element in acompensated image, settings of display elements in the compensated imagederived at least in part from the reference image information;processing the received compensation information to produce a set ofcoordinates indicating a corresponding off-grid location of the imageelement in the grid; calculating a value for the corresponding off-gridlocation of the image element based on a group of multiple elements inthe grid; and assigning the value to the image element in thecompensated image.

One or more embodiments herein include a computer readable storagemedium and/or system having instructions stored thereon. Theinstructions, when executed by computer processor hardware of one ormore computer devices, causes the computer processor hardware to performoperations of: receiving settings information for a first image, thefirst image including a set of multiple display elements; receivingmotion compensation information for a given display element in a secondimage, the motion compensation information indicating a coordinatelocation within a particular display element in the first image to whichthe given display element pertains; utilizing the coordinate location asa basis from which to select a grouping of multiple display elements inthe first image; and generating a setting for the given display elementin the second image based on settings of the multiple display elementsin the grouping.

The ordering of the steps has been added for clarity sake. These stepscan be performed in any suitable order.

Other embodiments of the present disclosure include software programs,firmware, and/or respective hardware to perform any of the methodembodiment steps and operations summarized above and disclosed in detailbelow.

Also, it is to be understood that the system, method, apparatus,instructions on computer readable storage media, etc., as discussedherein can be embodied strictly as a software program, as a hybrid ofsoftware, firmware, and/or hardware, or as hardware alone such as withina processor, or within an operating system or within a softwareapplication, etc.

As discussed above, techniques herein are well suited for use insoftware, firmware, and/or hardware applications that process signalsand produce bitstreams of encoded data, or that process bitstreams ofencoded data and produce renditions of signals. However, it should benoted that embodiments herein are not limited to use in suchapplications and that the techniques discussed herein are well suitedfor other applications as well.

Additionally, note that although each of the different features,techniques, configurations, etc., herein may be discussed in differentplaces of this disclosure, it is intended that each of the concepts canbe executed independently of each other or in combination with eachother. Accordingly, the one or more present inventions, embodiments,etc., as described herein can be embodied and viewed in many differentways.

Also, note that this preliminary discussion of embodiments herein doesnot specify every embodiment and/or incrementally novel aspect of thepresent disclosure or claimed invention(s). Instead, this briefdescription only presents general embodiments and corresponding pointsof novelty over conventional techniques. For additional details and/orpossible perspectives (permutations) of the invention(s), the reader isdirected to the Detailed Description section and corresponding figuresof the present disclosure as further discussed below.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other objects, features, and advantages of theinvention will be apparent from the following more particulardescription of preferred embodiments herein, as illustrated in theaccompanying drawings in which like reference characters refer to thesame parts throughout the different views. The drawings are notnecessarily to scale, with emphasis instead being placed uponillustrating the embodiments, principles, concepts, etc.

FIG. 1 is an example diagram describing a motion compensation systemleveraging non-limiting embodiments illustrated herein.

FIG. 2 is an example diagram describing motion compensation operationsaccording to non-limiting embodiments illustrated herein.

FIG. 3 shows two sample images that are object of motion estimation andcompensation, together with the corresponding information generated atencoding by a non-limiting embodiment illustrated herein.

FIG. 4A is an example diagram describing calculation of a referencelocation in the reference image according to non-limiting embodimentillustrated herein.

FIG. 4B illustrates a non-limiting example of the computations describedin FIG. 4A.

FIG. 5 is an example diagram describing on-the-fly resampling operationsaccording to non-limiting embodiments illustrated herein.

FIG. 6 shows a block diagram that implements motion compensationaccording to non-limiting embodiments illustrated herein.

FIG. 7 shows a block diagram of a computer system that provides dataprocessing according to the invention described herein.

FIG. 8 is an example diagram illustrating decoding of a signal accordingto embodiments herein.

FIGS. 9 and 10 are diagrams illustrating example methods according toembodiments herein.

DETAILED DESCRIPTION AND FURTHER SUMMARY OF EMBODIMENTS

Methods illustrated herein are suitable for any type ofmulti-dimensional signals, including without limitation sound signals,multichannel sound signals, pictures, two-dimensional images, videosignals, multi-view video signals, 3D video signals, volumetric signals,volumetric video signals, medical imaging signals, signals with morethan four dimensions, etc.

For simplicity, along the description the illustrated embodimentsusually adopt the use case of motion zones used in the context of motioncompensation operations for the encoding and decoding of videosequences, i.e., time-based signals consisting of a sequence of 2Dimages (commonly called “frames”, or “fields” in the case of interlacedvideo signals), with each element (in such non-limiting example casetypically referred to as “pixel”) being characterized by a set of colorsettings in a suitable color space (e.g., YUV, RGB, HSV, etc.).Different color planes (e.g., the luminance-Y plane and the twochrominance—U and V—planes) are often encoded separately, and often withdifferent resolutions (due to the lower sensitivity of the human eye tochrominance information), although the U and V planes typically leveragemotion compensation information calculated for the Y plane.

Methods and embodiments illustrated herein can be used in conjunctionwith one another and/or with other methods. Many of the preferredembodiments illustrated herein describe techniques and algorithms withthe goal of achieving compression, i.e., encoding a suitable renditionof the signal with a minimum quantity of bits. This also is anon-limiting example: other non-limiting embodiments achieve differentpurposes, such as robust and efficient filtering, image denoising,signal supersampling, machine vision, etc.

FIG. 1 is an example diagram describing a motion compensation systemwithin a signal processor configured as a decoder. The system isconfigured to reconstruct a target image 100-2 based on motioncompensation of a reference image 100-1 and on residual data.

Motion Compensation block 120 receives Reference Image 100-1 andInformation on Motion Compensation 110, the information 110 specifyinghow to leverage Reference Image 100-1 in order to generate a suitableprediction for Target Image 100-2.

Based at least in part on Reference Image 100-1 and on information 110,Motion Compensation block 120 generates Destination Image 130, i.e., amotion-compensated image which serves as a prediction for Target Image100-2.

Combiner 150 combines Destination Image 130 with Residual Data 140,reconstructing a rendition of Target Image 100-2.

Novel methods described herein illustrate innovative approaches toperform the function of Motion Compensation block 120.

FIG. 2 is an example diagram describing motion compensation operationsaccording to a non-limiting embodiment. For each element of theDestination Image 130, Motion Compensator 205 generatesmotion-compensated settings (i.e., values that are based on motioncompensation of a suitable location of the reference image), accordingto the steps illustrated in the diagram.

First of all, New Coordinates Calculator 220 receives the coordinates(x, y) 200 of the given element in the Destination Image and DescriptiveInformation on Motion 210, calculating Coordinates (x¹, y¹) 230.Coordinates (x¹, y¹) represent the location in the Reference Image thatcorrespond to location (x, y) of the Destination Image. According to theinvention, x¹ and y¹ are expressed with a continuous coordinates system(e.g., by means of floating point numbers, or high-precision fixed pointnumbers), essentially allowing to indicate an arbitrary location of theReference Image, regardless of the actual resolution (i.e., samplinggrid) of the Reference Image.

Since (x¹, y¹) are not necessarily identifying an element of theReference Image (which would be characterized by integer coordinates),Motion Compensator 205 cannot directly fetch the value of the ReferenceImage in location (x¹, y¹). In the described non-limiting embodiment, anon-the-fly resampling method is used, with Resampler 260 receivingReference Image 100-1, Metadata 245 on the Reference Image (if any) andInformation 250 on the resampling method to leverage, thus generatingthe value of the Reference Image in location (x¹, y¹), i.e., thespecific settings 270 to use for coordinates (x, y) of the DestinationImage.

Resampler 260 performs on-the-fly filtering (i.e., after having receivedcoordinates 230) of the elements of Reference Image 100-1 surroundingthe specified arbitrary location with continuous coordinates (x¹, y¹).The filtering is performed according to a suitable kernel, by selecting,based on Information on the resampling method 250, among a Lanczosfilter, a cubic filter or a linear filter (in absence of Information250, Resampler 260 selects a default kernel). In this way, for a decoderimplementing motion compensation with Motion Compensator 205 it is notnecessary to compute and store a supersampled rendition of ReferenceImage 100-1, and select samples from the finite grid of suchsupersampled rendition. The decoder may just store a reference imagewith normal sample grid (i.e., the same resolution as the destinationimage) and select any arbitrary position, generating/interpolating onthe fly the corresponding sample value in the reference image for thearbitrary position. In some situations such method allows forsignificant reduction of memory requirements at the decoder, and at thesame time also allows for more precise motion compensation (withcorresponding compression benefits). In other situations the method alsoallows for savings in terms of processing power: the decoder justproduces the supersampled reference samples that it needs in thespecific positions where it needs them, not having to generate—eitherfor the whole image or for portions of the image—a whole supersampledreference grid (i.e., also calculating many samples that will never beaccessed, leveraged or displayed).

FIG. 3 shows two sample images that are object of motion estimation andcompensation, together with the corresponding information generated atencoding by a non-limiting embodiment illustrated herein.

In particular, the diagram illustrates two subsequent images 100-1 and100-2 in a sequence of images. The images show significant correlationbetween each other, and such correlation can be enhanced even further bymeans of motion compensation: the background slightly rotatescounter-clockwise, and an object 301-1 in image 100-1 moves to becomeobject 301-2 in image 100-2 (wherein the movement involves bothtranslation toward the lower right and de-zoom, i.e., the object isbecoming smaller). An object 302-2, not visible in image 100-1, becomesvisible in image 100-2.

A non-limiting embodiment is able, by starting from the images 100-1(reference image) and 100-2 (target image), to encode/decode a targetimage by using motion zones having arbitrary shapes (e.g., contiguous oreven non-contiguous), wherein the shapes are chosen on the basis ofimages 100-1 and 100-2. The shape of each motion zone can be arbitrary,and is not limited to rectangular blocks. In some situations this mayallow to increase the efficiency and the effectiveness of motioncompensation, more closely following the borders of objects (e.g.,avoiding to “drag” elements that are close to the borders of a movingobject) and more efficiently transmitting accurate motion informationfor each given element of an image.

More specifically, by processing images 100-1 and 100-2, an encoderidentifies motion zone 311-2 (representing the portion of the backgroundof image 100-2 that can be efficiently predicted by motion-compensatingelements of reference image 100-1), motion zone 312-2 (representingobject 301-2 of image 100-2, which can be efficiently predicted bymotion-compensating object 301-1 in image 100-1) and a non-contiguousresidual motion zone made of 313-2, 314-2 and 315-2 (representing theelements of image 100-2 that were not visible in image 100-1, andconsequently cannot be predicted by motion-compensating elements ofimage 100-1).

Encoder 100 also generates descriptive information 320-2 associated tomotion zones. In a non-limiting embodiment, such information comprises aZRO (Zoom, Rotation and Offset) motion matrix for each motion zone, anda default parameter for the residual motion zone (“N/A motion”, meaningthat the elements of the residual motion zone cannot be suitablypredicted through motion compensation).

The detection of motion zones and the generation of descriptiveinformation associated to motion zones is performed according tosuitable motion zone detection methods, not described in thisapplication.

Lastly, the encoder generates residual data 330-2 to adjust theprediction of the target image 100-2 obtained by means ofmotion-zone-based motion compensation of reference image 100-1. Combinedwith the prediction obtained by motion compensation, residual data allowto fully reconstruct a rendition of target image 100-2.

A method for decoding images encoded by using the above-describedencoding method comprises the following phases:

-   -   a. receiving at least one motion zone and one set of descriptive        information on motion;    -   b. receiving a first (reference) image;    -   c. computing a motion-compensated prediction for the second        (target) image by using the first image received during phase b,        the motion zones and the descriptive information on motion        received during phase a;    -   d. receiving and decoding residual data (if any);    -   e. combining the prediction for the second image with residual        data (if any), producing a rendition of the second image.

One or more of the phases can occur concurrently, without being strictlysequential.

FIG. 4A is an example diagram describing calculation of a referencelocation in the reference image, illustrating a non-limiting embodimentof New Coordinate Calculator 220.

Homogenous Coordinate Converter 400 receives coordinates (x, y) of theelement in the Destination Image and converts them into homogenouscoordinates (x, y, 1) 405. Matrix Calculator 410 receives DescriptiveInformation on Motion 210 and converts it into Motion Matrix M 415.Multiplier 420 receives homogenous coordinates 405 and motion matrix415, and calculates Transformed Coordinates 425. If Motion Matrix M 415was an affine transform, the third coordinate of Transformed Coordinates425 would already be equal to one, otherwise (e.g., if matrix M was aprojective matrix) it is necessary to normalize coordinates 425.Homogenous Coordinates Converter 430 performs such normalization (ifnecessary), generating homogenous coordinates (x¹, y¹, 1). Lastly,Selector 440 generates Coordinates (x¹, y¹) in the Reference Image.

Coordinates (x, y) of the element in the Destination Image describe anelement of the sample grid of the Destination Image (i.e., they describeinteger positions of the sample grid), while Coordinates (x¹,y¹)—expressed with fractional numbers—may describe any arbitrarylocation of the Reference Image (comprising, without limitation,locations outside of the boundaries of the sample grid of the ReferenceImage). Coefficients of Matrix M 415 may be fractional numbers, andcalculations performed within New Coordinate Calculator 220 comprisecalculations with fractional numbers (e.g., without limitation, floatingpoint operations, or operations with high-precision fixed pointnumbers).

FIG. 4B illustrates a non-limiting example embodiment of thecomputations described in FIG. 4A. The non-limiting example shows a casein which Motion Matrix M 415 corresponds to an affine transform, so thatthe multiplication of (x, y, 1) times Matrix M generates a set ofcoordinates (x¹, y¹, 1) that is already normalized.

In a non-limiting embodiment, when coordinates 200 of element 465 andmatrix 415 are such that coordinates 230 of position 455 are outside ofthe reference image beyond a threshold (e.g., without limitation,outside of the sampling grid of the reference image by an amount higherthan the distance between two neighbouring elements), element 465 isassigned a default value (e.g., “N/A” value).

FIG. 5 is an example diagram describing on-the-fly resampling operationsaccording to a non-limiting embodiment.

Position 455 with coordinates (x¹, y¹) is not described by any elementof the sample grid of Reference Image 100-1. As a consequence, no valueis readily available for the corresponding motion-compensated element ofDestination Image 130. However, it is possible to calculate such a valueon the fly, by selecting a suitable set of elements (“predictors”) closeto coordinates (x¹, y¹) and by performing a weighted average of theirvalues according to suitable weights (e.g., a suitable filteringkernel). FIG. 5 illustrates a non-limiting example where a bicubicfilter kernel is used, and consequently the value of Reference Image100-1 in position (x¹, y¹) is calculated by performing a weightedaverage of 16 predictor elements.

In order to perform the resampling, Resampler 260 must identify the 16predictors to leverage for the weighted average (i.e., where to“position the filter”) and must compute the specific weights to be used.The integer component of the (x¹, y¹) coordinates indicate where toposition the filter, and the fractional component of the coordinatesindicate what coefficients to use in the filter.

Floor Calculator 500 receives Coordinates (x¹, y¹) 230, calculating theinteger components 505 of coordinates (x¹, y¹). In the non-limitingexample of coordinates based on an origin at the top left of thereference image, the Integer Coordinates 505 identify the closestelement of the sample grid at the top left of coordinates (x¹, y¹).Selector of Predictors 510, which receives Reference Image 240, Metadataon Reference Image 245 (if any) and Info on Resampling Method 250 (ifany), is thus able to select the 16 predictors, by selecting the 4×4block of elements of the reference image that has the element withcoordinates 505 as the second element in the diagonal from the top leftof the block. If some of the 16 predictors are assigned “N/A” (notavailable value, e.g., because they are outside of the boundaries of thereference image), they are taken out of the set of predictors, and onlythe predictors with a value are kept.

Calculator of Weights 520 receives coordinates (x¹, y¹) and thecoordinates of the predictors. Based on the chosen resampling method(e.g., in this non-limiting example, bicubic filtering) and on thefractional component of coordinates (x¹, y¹)—i.e., on the distancebetween position (x¹, y¹) and the element with integer coordinates 505—

Calculator of Weights 520 calculates the weights to apply to eachpredictor. If less than 16 predictors have been selected, Calculator ofWeights 520 generates the suitable bicubic weights for the ideal 4×4kernel, then selects only the coefficients corresponding to thepredictors that were selected and normalizes the cofficients so thattheir sum is equal to 1.

Finally, Multiplier 530 receives both the predictors and the weight toapply to each predictor, consequently calculating the resampled value ofReference Image in position (x¹, y¹), i.e., Settings 270 for coordinates(x, y) in the destination image.

In a non-limiting embodiment, in order to avoid the need to generate inreal time the precise coefficients of the filter based on the specific(x¹, y¹) coordinates, the signal processor implementing Calculator ofWeights 520 is configured so as to access suitable lookup tables with agiven number of pre-calculated sets of coefficients (e.g., in anon-limiting example, 128 sets of coefficients), and then to interpolatethe coefficients of the two sets of coefficients that are closest to thefractional components of the specific (x¹, y¹) coordinates.

In a non-limiting embodiment, metadata 245 on reference image containsinformation on modifications to apply to Reference Image 100-1 beforemotion compensation (e.g., without limitation, a scale parameter toapply to values of the image, so as to account for changes in lighting).

FIG. 6 shows a block diagram that implements motion compensationaccording to a non-limiting embodiment leveraging motion zones. Inparticular, Motion Compensator 600 generates a prediction of the targetimage by means of motion compensation with continuous coordinates.

Block 600 receives information comprising a first set of input 610-0, .. . , 610-n, and 620-0, . . . , 620-n for motion zones and descriptiveinformation on motion, a second input 200 for element coordinates (x, y)in the destination image (i.e., identifying the specific element topredict, according to the sample grid of the destination image), a thirdinput 240 for the reference image, and an output 270 for the values toassign to the motion compensated element (i.e., the settings forcoordinates (x, y) in the destination image). Based on elementcoordinates 200, Motion Compensator 600 identifies the motion zone(s) towhich the element belongs, and—based at least in part on thecorresponding descriptive information on motion and on elementcoordinates 200—generates the location (x¹, y¹) in reference image240—specified with a continuous coordinate system, independently of thesample grid of the reference image and assuming that it is possible tofetch a value for any arbitrary location in the reference image—thatcorresponds to element coordinates 200 in the destination image. Lastly,based on reference image 240 and on the generated location (x¹, y¹),motion compensator 600 calculates the motion-compensated settings 270(i.e., the prediction) for the specific element in position (x, y) ofthe sample grid of the destination image.

FIG. 7 is a block diagram of a computer system 800 that providescomputer processing according to embodiments herein.

Computer system 800 can be or at least include a computerized devicesuch as a personal computer, processing circuitry, television, playbackdevice, encoding device, workstation, portable computing device, mobilephone, tablet, console, set top box, network terminal, processingdevice, network device, operating as a switch, router, server, client,etc.

Note that the following discussion provides a basic embodimentindicating how to carry out functionality associated with a signalprocessor as previously discussed.

However, it should be noted that the actual configuration for carryingout the operations as described herein can vary depending on arespective application.

As shown, computer system 800 of the present example comprises acommunication bus 811, which provides communication with a computerreadable storage media 812 such as a non-transitory computer-readablestorage medium, etc., in which digital information can be stored andretrieved.

Computer system 800 can further comprise a processor 813, an I/Ointerface 814, and a communications interface 817.

In one embodiment, I/O interface 814 provides connectivity to repository180, and if present, to a screen display, audio speakers, peripheraldevices 816 such as a keyboard, a computer mouse, etc.

As briefly mentioned above, the computer readable storage media 812 canbe any suitable device and/or hardware such as memory, optical storage,solid state storage, hard drive, floppy disk, etc. In one embodiment,the storage media 812 is a non-transitory storage media (i.e.,non-carrier wave media) configured to store instructions associated witha signal processor application 840-1. The instructions are executed by arespective resource such as the processor 813 in order to perform any ofthe operations as discussed herein.

The communications interface 817 enables computer system 800 tocommunicate over a network 190 in order to retrieve information fromremote sources and communicate with other computers, switches, clients,servers, etc. The I/O interface 814 also enables processor 813 toretrieve or attempt retrieval of stored information from repository 180.

As shown, the computer readable storage media 812 can be encoded withsignal processor application 840-1 executed by processor 813 as aprocess 840-2.

Note that the computer system 800 can be embodied to include a computerreadable storage media 812 for storing data and/or logic instructions.

The code associated with the signal processor application 840-1 canimplement the functionalities of the signal processors illustratedherein and/or other resources necessary to implement the inventiondiscussed herein.

During operation of one embodiment, processor 813 accesses computerreadable storage media 812 via the communication bus 811 in order tolaunch, run, execute, interpret or otherwise perform the instructions ofsignal processor application 840-1 stored in computer readable storagemedia 812. Execution of the signal processor application 840-1 producesprocessing functionality in processor 813. In other words, the signalprocessor process 840-2 associated with processor 813 represents one ormore aspects of executing signal processor application 840-1 within orupon the processor 813 in the computer system 800.

FIG. 8 is an example diagram illustrating use of a reference image toderive settings for image elements in a compensated image according toembodiments herein.

More specifically, as shown, computer processor hardware such as adecoder receives settings information specifying settings of displayelements 810 (e.g., display elements 810-1, 810-2, 810-3, . . . ) inrendition of image 840 (such as a reference image). The settings canindicate any suitable parameter such as color, chrominance, luminance,etc.

One or more of the display elements in the rendition of image 840 canrepresent at least a portion of an object present in the referenceimage. The rendition of image 850 can represent another image in asequence that includes a portion of the object captured in the referenceimage. As discussed herein, the reference image (rendition of image 840)can be used as a basis to reconstruct another image such as rendition ofimage 850.

In this non-limiting example embodiment, rendition of image 850 such asa motion compensated image includes multiple display elements 820 (e.g.,display element 820-1, display element 820-2, display element 820-3, andso on). As previously discussed, the rendition of image 850 can begenerated based on a rendition of an object in reference image. Motionvector 860 specifies a precise location in the rendition of image 840 towhich the display element 820-9 pertains. According to embodimentsherein, as an alternative to upsampling the rendition of image 840 intoa supersampled (i.e., super high resolution) image as in conventionalmethods, to accommodate sub-element or fractional element motioncompensation for one or more elements, embodiments herein includederiving a setting for display element 820-9 based on a relatively lowlevel of quality or low resolution rendition of image 840.

As a specific example, the computer processor hardware receives motioncompensation information and/or other metadata or reconstruction datafor display element 820-9 in rendition of image 850. Rendition of image850 represents an image being created based at least in part on settingsof display elements in rendition of image 840 (reference image).Received motion compensation information indicates a coordinate locationwithin a display element 810-11 in the rendition of image 840 to whichthe display element 820-9 pertains. The received reconstruction data toreconstruct the settings for display element 820-9 can specify which ofone or more reference images as well as one or more locations withineach reference image are to be used as a basis to reconstruct arespective setting for the display element. In a simple exampleembodiment, motion compensation information indicates a location withina single reference image to use as a basis to determine settings for thedisplay element 820-9.

In one embodiment, computer processor hardware converts the receivedcompensation information into motion vector 860 (assigned to displayelement 820-9) specifying an off-grid location in rendition of image 840(grid) to which the element 820-9 pertains. The location of displayelement 820-9 is off-grid in rendition of image 840 because it does notfall directly over only a single display element in rendition of image840. In this example embodiment, the location of image element 820-9 asspecified by the motion vector 860 at least partially overlaps multipledisplay elements 810-6, 810-7, 810-10, and 810-11 in rendition of image840.

The computer processor hardware utilizes the coordinate location asspecified by the motion vector 860 as a basis from which to select agrouping of multiple display elements in the rendition of image 840. Forexample, in this example embodiment, due to proximity or other suitableparameter, assume that the computer processor hardware selects displayelements 810-6, 810-7, 810-10, 810-11, 810-12, and 810-15 as suitablenearby elements from which to generate one or more settings for displayelement 820-9. Based on such settings of selected display elements810-6, 810-7, 810-10, 810-11, 810-12, and 810-15, the computer processorhardware generates settings for display element 820-9.

In one embodiment, the computer processor hardware calculates settinginformation for display element 820-9 based on rendition of image 840 asif display element 820-9 were located in the fractionally offsetlocation as specified by the motion vector 860. For example, thecentroid of display element 820-9 as specified by the motion vector 860does not align exactly over the centroid of display element 810-11.Embodiments herein include generating a setting for the display element820-9 in the second image based on settings of the multiple displayelements 810-6, 810-7, 810-10, 810-11, 810-12, and 810-15 in theselected grouping.

Note that any suitable algorithm amongst multiple algorithms can be usedto derive the setting for the display element 820-9 based on theselected grouping of display elements. For example, assume in thisexample that received reconstruction information associated with displayelement 820-9 indicates to the computer processor hardware to use aparticular algorithm to generate the grouping to include displayelements 810-6, 810-7, 810-10, 810-11, 810-12, and 810-15 as mentionedabove.

In accordance with a specified algorithm, the computer processorhardware calculates the setting for the display element 820-9 based atleast in part on distances of the multiple display elements in thegrouping with respect to the coordinate location (of the centroid ofdisplay element 820-9 as specified by the motion vector 860) in theparticular display element 810-11.

For example, the length of line L1 represents a distance betweencentroid (i.e., center) of display element 810-6 (as depicted byrespective symbol x in center of display element 810-6) and a centroidof display element 820-9 (as specified by the motion vector 860 and asdepicted by a dot in display element 820-9); the length of line L2represents a distance between centroid of display element 810-7 (asdepicted by respective symbol x in center of display element 810-7) anda centroid of display element 820-9 (as specified by the motion vector860 and as depicted by a dot in display element 820-9); the length ofline L3 represents a distance between centroid of display element 810-12(as depicted by respective symbol x in center of display element 810-12)and a centroid of display element 820-9; the length of line L4represents a distance between centroid of display element 810-11 (asdepicted by respective symbol x in center of display element 810-11) anda centroid of display element 820-9; the length of line L5 represents adistance between centroid of display element 810-15 (as depicted byrespective symbol x in center of display element 810-15) and a centroidof display element 820-9; the length of line L6 represents a distancebetween centroid of display element 810-10 (as depicted by respectivesymbol x in center of display element 810-10) and a centroid of displayelement 820-9.

As mentioned, computer processor hardware can be configured to generatea value such as a display setting in any suitable manner. For example,the algorithm can be configured to generate the setting for displayelement 820-9 based on calculating a average of settings associated withnearby display elements 810-6, 810-7, 810-10, 810-11, 810-12, and810-15. For example, assume that display element 810-6 is set to a valueof 24; display element 810-7 is set to a value of 60; display element810-10 is set to a value of 228; display element 810-11 is set to avalue of —123; display element 810-12 is set to a value of —41; anddisplay element 810-15 is set to a value of 2. The average of thesevalues is 25. Accordingly, in this instance, the algorithm assigns thedisplay element 820-9 a value of 25.

In accordance with further embodiments, the algorithm can be configuredto generate weights for each of the display elements in the selectedgrouping. For example, in a non-limiting embodiment the selectedalgorithm to generate settings for display element 820-9 can beconfigured to calculate a length of each of the lines L1, L2, L3, L4,L5, and L6 and generate a corresponding weight for a respective displayelement depending on a length of a line. The shorter the line, thehigher the magnitude of the respective weight such that the settingcalculated for the display element 820-9 is more heavily weighted basedon the setting of nearer elements than farther elements. As a specificexample, the length of line L1 can be 7 units; the length of line L2 canbe 5 units; the length of line L3 can be 6 units; the length of line L4can be 1 unit; the length of line L5 can be 6 units; the length of lineL7 can be 4 units. Based on lengths, the algorithm sets W1 for line L1to be 0.053; the algorithm sets W2 for line L2 to be 0.157; thealgorithm sets W3 for line L3 to be 0.105; the algorithm sets W4 forline L4 to be 0.368; the algorithm sets W5 for line L5 to be 0.105; thealgorithm sets W6 for line L6 to be 0.21. The sum of weights can equalone. To calculate the setting for display element 820-9 based onweights, the algorithm generates the value of display element 820-9 asfollows:

-   Value=(W1×S1)+(W2×S2)+(W3×S3)+(W4×S4)+(W5×S5)+(W6×S6);    -   where S1=the setting of display element 810-6=24;        -   S2=the setting of display element 810-7=60;        -   S3=the setting of display element 810-10=228;        -   S4=the setting of display element 810-11=−123;        -   S5=the setting of display element 810-12=−41;        -   S6=the setting of display element 810-15=2.-   Value=(0.053×24)+(0.157×60)+(0.105×228)+(0.368×(−123))+(0.105×(−41))+(0.21×2)=−14.2;

In this example instance, the algorithm assigns the setting of displayelement 820-9 to a value of −14.2.

In accordance with further non-limiting embodiments, the algorithm canbe configured to generate weights for each of the display elements inthe selected grouping according to suitable resampling kernels (such asa bilinear kernel, a bicubic kernel, a lanczos kernel, etc.). In one ofsuch embodiments, a signal processor identifies the element 810-11 inthe reference image that is closest to the coordinate location, and thenselects a grouping of 16 elements comprising said element. A lookuptable is available with several possible values of the kernel weights,according to the distance of the coordinate location from the centroidof element 810-11. In an embodiment, the signal processor calculates theweights by interpolating sets of weights in the lookup tablecorresponding to locations that are close to the actual x-distance andy-distance of the coordinate location from the centroid of element810-11.

Accordingly, embodiments herein can include producing weighted valuesthat vary depending on a respective distance between the centroid of thegiven display element as specified by the motion vector 860 and arespective centroid of an element in the grouping in the referenceimage; and applying the weighted values to settings of the displayelements in the grouping to produce the setting for the display element820-9.

Thus, assuming an unlimited or substantially high precision of themotion vector, embodiments herein are able to reconstruct images basedon fractional movements of objects from one image plane to the next. Inother words, a reference image in a sequence of multiple video framesmay include a representation of an object. The reference image (atnormal resolution) can be used as a basis to produce a fractionallydisplaced representation of the moving object in a next frame. In otherwords, a subsequent frame can include a rendition of the moving objectin the reference frame as being moved from one frame to the next by lessthan a full pixel or at a sub-pixel level. In other non-limitingembodiments described in related applications, the reference image maynot be a precedent or subsequent image in a sequence of images, but mayinstead be a support image generated specifically to support thereconstruction of multiple time-correlated images. In some of suchembodiments, the reference image may even be a downblended predictor ina tiered temporal hierarchy, with a spatial resolution lower than theresolution of the motion-compensated image.

As discussed above, the motion vector 860 can specify a coordinatelocation other than a center of the display element 810-11 in therendition of image 840. The motion vector 860 can be of any suitableresolution enabling the corresponding display element 820-9 to refer toany location within the rendition of image 840. Thus, the motion vector860 can be an offset value or vector pointing to any location within arespective element or cell in the rendition of image 840 such as areference image.

In one embodiment, the specific coordinate location (such as a locationof centroid of display element 820-9) in rendition of image 840 asspecified by the motion vector 860 can be offset with respect to acentroid of the display element 810-11 by a fractional value of adisplay element or pixel, such as a fractional value greater than zerobut less than 1. As mentioned, the rendition of image 850 can include arepresentation of the object in reference image, but displaced by lessthan a full pixel with respect to the reference image.

As shown, the resolution of the rendition of image 840 such as areference image can be substantially equal to a resolution of therendition of image 850 (e.g., an image being rendered based at least inpart on the reference image).

Note that, in accordance with further embodiments, the resolution of therendition of image 840 can be different than the resolution of therendition of image 850. For example, in one embodiment, the resolutionof the rendition of the first image 840 can be substantially differentthan a resolution of the rendition of second image 850.

Accordingly, the coordinate location as specified by the motion vector860 can be used as a basis from which to select a grouping of multipledisplay elements in a first image such as rendition of image 840. Asmentioned, the computer processor hardware can be configured to create agrouping of multiple display elements to include display element 810-11and one or more other display elements in the rendition of image 840,the one or more other display elements located in a vicinity of theparticular display element 810-11.

In one non-limiting example embodiment, the computer processor hardwareinterpolates the setting of the display element 820-9 based at least inpart on: i) a proximity of the coordinate location (as specified bymotion vector 860) with respect to a centroid of the particular displayelement 810-11, ii) a setting of the particular display element 810-11,and iii) settings of the display elements located in the vicinity of theparticular display element 810-11.

Note that the process of motion compensation can be applied to each ofmultiple elements in a compensated image to derive a rendition of image850 based on rendition of image 840. For example, embodiments hereininclude generating a respective motion vector for each of multipledisplay elements in the rendition of image 850. In a manner aspreviously discussed, the computer processor hardware reproducing therendition of image 850 utilizes the respective motion vector associatedwith a corresponding display element in the rendition of image 850 toidentify a specific location in the rendition of image 840 to which thecorresponding display element pertains. The computer processor hardwarethen uses the specific location as specified by the motion vector 860 asa basis in which to identify neighboring display elements in thereference image and generate a setting for the corresponding displayelement.

Functionality supported by the different resources will now be discussedvia flowcharts in FIGS. 9-10. Note that the steps in the flowchartsbelow can be executed in any suitable order.

FIG. 9 is a flowchart 900 illustrating an example method according toembodiments. Note that there will be some overlap with respect toconcepts as discussed above.

In processing block 910, computer processor hardware (such as decoderprocessing hardware) receives reference image information or referenceimage. The reference image information such as rendition of image 840defines a grid of multiple elements at a given resolution.

In processing block 920, the computer processor hardware receivescompensation information for an image element such as display element820-9 in a compensated image such as rendition of image 850. Thesettings of display elements in the compensated image are derived atleast in part from the reference image information.

In processing block 930, computer processor hardware processes thereceived compensation information to produce a set of coordinates suchas a location as specified by the motion vector 860 indicating acorresponding off-grid location of the image element in the grid. Asmentioned, the location of the display element 820-9 does not fallwithin grid lines, but is at least partially overlaid on multipledisplay elements.

In processing block 940, computer processor hardware calculates a valuefor the corresponding off-grid location of the image element based on agroup of multiple elements in the grid.

In processing block 950, computer processor hardware assigns thecalculated value to the image element in the compensated image.

FIG. 10 is a flowchart 1000 illustrating an example method according toembodiments. Note that there will be some overlap with respect toconcepts as discussed above.

In processing block 1010, computer processor hardware receives settingsinformation for a first image such as a reference image. The first imageincludes a set of multiple display elements.

In processing block 1020, computer processor hardware receive motioncompensation information for a given display element in a second image.The motion compensation information indicates a coordinate locationwithin a particular display element in the first image to which thegiven display element pertains.

In processing block 1030, computer processor hardware utilizes thecoordinate location as a basis from which to select a grouping ofmultiple display elements in the first image.

In processing block 1040, computer processor hardware generates asetting for the given display element in the second image based onsettings of the multiple display elements in the grouping.

Note again that techniques herein are well suited for encoding anddecoding in hierarchical systems. However, it should be noted thatembodiments herein are not limited to use in such applications and thatthe techniques discussed herein are well suited for other applicationsas well.

Based on the description set forth herein, numerous specific detailshave been set forth to provide a thorough understanding of claimedsubject matter. However, it will be understood by those skilled in theart that claimed subject matter may be practiced without these specificdetails. In other instances, methods, apparatuses, systems, etc., thatwould be known by one of ordinary skill have not been described indetail so as not to obscure claimed subject matter. Some portions of thedetailed description have been presented in terms of algorithms orsymbolic representations of operations on data bits or binary digitalsignals stored within a computing system memory, such as a computermemory. These algorithmic descriptions or representations are examplesof techniques used by those of ordinary skill in the data processingarts to convey the substance of their work to others skilled in the art.An algorithm as described herein, and generally, is considered to be aself-consistent sequence of operations or similar processing leading toa desired result. In this context, operations or processing involvephysical manipulation of physical quantities. Typically, although notnecessarily, such quantities may take the form of electrical or magneticsignals capable of being stored, transferred, combined, compared orotherwise manipulated. It has been convenient at times, principally forreasons of common usage, to refer to such signals as bits, data, values,elements, symbols, characters, terms, numbers, numerals or the like. Itshould be understood, however, that all of these and similar terms areto be associated with appropriate physical quantities and are merelyconvenient labels. Unless specifically stated otherwise, as apparentfrom the following discussion, it is appreciated that throughout thisspecification discussions utilizing terms such as “processing,”“computing,” “calculating,” “determining” or the like refer to actionsor processes of a computing platform, such as a computer or a similarelectronic computing device, that manipulates or transforms datarepresented as physical electronic or magnetic quantities withinmemories, registers, or other information storage devices, transmissiondevices, or display devices of the computing platform.

In accordance with different embodiments, note that computer system maybe any of various types of devices, including, but not limited to, apersonal computer system, desktop computer, laptop, notebook, netbookcomputer, mainframe computer system, handheld computer, workstation,network computer, application server, storage device, a consumerelectronics device such as a camera, camcorder, set top box, mobiledevice, smartphone, tablet, video game console, handheld video gamedevice, a peripheral device such as a switch, modem, router, or, ingeneral, any type of computing or electronic device.

While embodiments herein have been particularly shown and described withreferences to preferred embodiments thereof, it will be understood bythose skilled in the art that various changes in form and details may bemade therein without departing from the spirit and scope of the presentapplication as defined by the appended claims. Such variations areintended to be covered by the scope of this present application. Assuch, the foregoing description of embodiments of the presentapplication is not intended to be limiting. Rather, any limitations tothe invention are presented in the following claims.

We claim:
 1. A method comprising: via computer processor hardware,performing operations of: utilizing location information to identify acorresponding location of a particular image element in a referenceimage; identifying a group of multiple image elements in the referenceimage associated with the particular image element, each of the multipleimage elements in the group being on-grid in the reference image; andproducing a value for the particular image element based on settings ofthe multiple image elements in the group.
 2. The method as in claim 1,wherein the corresponding location of the particular image element isoff-grid with respect to multiple image elements residing in a grid ofthe reference image
 3. The method as in claim 2 further comprising:assigning the value to a first image element of an image rendition beingreproduced based on the reference image.
 4. The method as in claim 3,wherein the location information includes a motion vector indicating arelationship of the first image element in the image rendition to theparticular image element in the reference image.
 5. The method as inclaim 4, wherein the first image element is one of multiple imageelements of the image rendition being reproduced, the first imageelement of the image rendition being an on-grid image element of theimage rendition being reproduced.
 6. The method as in claim 3, wherein aresolution of the image rendition being reproduced is different than aresolution of the reference image.
 7. The method as in claim 1, whereinproducing the value for the particular image element includes applyingweight values to settings of the multiple image elements in the group.8. The method as in claim 1 further comprising: using the correspondinglocation of the particular image element to identify the group ofmultiple image elements in the reference image that pertain to theparticular image element.
 9. The method as in claim 1, wherein thelocation information indicating the corresponding location of theparticular image element is expressed with a substantially higherresolution than a resolution of a grid defining the reference image. 10.The method as in claim 1, wherein producing the value for the particularimage element includes: calculating the setting for the particular imageelement based at least in part on distances of the multiple imageelements in the group with respect to the corresponding location of theparticular image element in the reference image.
 11. The method as inclaim 1, wherein producing the value for the particular image elementincludes: producing weighted values, the weighted values varyingdepending on a distance between a centroid of the particular imageelement and a respective centroid of an image element in the group; andapplying the weighted values to settings of the image elements in thegroup to produce the value for the particular image element.
 12. Themethod as in claim 1, wherein the group of multiple elements in thereference image represents at least a portion of an object present in animage rendition being reproduced from the reference image.
 13. Themethod as in claim 1, wherein identifying the group of multiple imageelements in the reference image comprises: selecting the group ofmultiple image elements based at least in part on a respective nearnessof each of the multiple image elements with respect to the particularimage element.
 14. The method as in claim 13, wherein the multiple imageelements in the group includes a first display element and a seconddisplay element; and wherein producing the value for the particularimage element comprises: i) producing a first length value, the firstlength value representing a respective length between a coordinatelocation of the particular image element and the first display element,and ii) producing a second length value, the second value representing arespective length between the coordinate location of the particularimage element and the second display element.
 15. The method as in claim1, wherein an overlay coverage region of the particular image element inthe reference image overlaps at least a portion of the multiple imageelements in the group.
 16. The method as in claim 1, wherein at leastone of the multiple image elements in the group resides completelyoutside of an overlay coverage region of the particular image element inthe reference image.
 17. The method as in claim 1 further comprising:applying unique weight values to settings of the multiple image elementsin the group to produce the value for the particular image element inthe reference image.
 18. Computer-readable hardware storage havinginstructions stored thereon, the instructions, when carried out bycomputer processor hardware, cause the computer processor hardware toperform operations of: utilize motion vector information to identify acorresponding location of a particular image element in a referenceimage; identify a group of multiple image elements in the referenceimage associated with the particular image element, each of the multipleimage elements in the group being on-grid in the reference image; andproduce a value for the particular image element based on settings ofthe multiple image elements in the group.
 19. The computer-readablehardware storage as in claim 1, wherein the motion vector informationindicates a relationship of a first image element in an image renditionbeing reproduced to the particular image element in the reference image;and wherein the instructions cause the computer processor hardware to:assign the value to a first image element of the image rendition beingreproduced based on the reference image.
 20. A computer systemcomprising: computer processor hardware; and a hardware storage resourcecoupled to the computer processor hardware, the hardware storageresource storing instructions that, when executed by the computerprocessor hardware, cause the computer processor hardware to: utilize aset of coordinates to identify a corresponding location of a particularimage element in a reference image; identify a group of multiple imageelements in the reference image associated with the particular imageelement, each of the multiple image elements in the group being on-gridin the reference image; and produce a value for the particular imageelement based on settings